Statistical Analysis of Some Multi-Category Large Margin Classification Methods

The purpose of this paper is to investigate statistical properties of risk minimization based multicategory classification methods. These methods can be considered as natural extensions of binary large margin classification. We establish conditions that guarantee the consistency of classifiers obtained in the risk minimization framework with respect to the classification error. Examples are provided for four specific forms of the general formulation, which extend a number of known methods. Using these examples, we show that some risk minimization formulations can also be used to obtain conditional probability estimates for the underlying problem. Such conditional probability information can be useful for statistical inferencing tasks beyond classification. 1. Motivation Consider a binary classification problem where we want to predict label y∈ {±1} based on observation x. One of the most significant achievements for binary classification in machine learning is the invention of large margin methods, which include support vector machines and boosting algorithms. Based on a set of training samples (X1,Y1), . . . ,(Xn,Yn), a large margin binary classification algorithm produces a decision function f̂ (·) by minimizing an empirical loss function that is often a convex upper bound of the binary classification error function. Given f̂ (·), the binary decision rule is to predict y = 1 if f̂ (x)≥ 0, and to predict y =−1 otherwise (the decision rule at f̂ (x) = 0 is not important). In the literature, the following form of large margin binary classification is often encountered: we minimize the empirical risk associated with a convex function φ in a pre-chosen function class Cn that may depend on the sample size: f̂ (·) = arg min f (·)∈Cn 1 n n ∑ i=1 φ( f (Xi)Yi). (1) Originally such a scheme was regarded as a compromise to avoid computational difficulties associated with direct classification error minimization, which often leads to an NP-hard problem. Some recent works in the statistical literature argued that such methods could be used to obtain conditional probability estimates. For example, see Friedman et al. (2000), Lin (2002), Schapire and Singer (1999), Zhang (2004), Steinwart (2003) for related studies. This point of view allows people to show the consistency of various large margin methods: that is, in the large sample limit, the obtained classifiers achieve the optimal Bayes error rate. For example, see Bartlett et al. (2003), Jiang (2004), Lugosi and Vayatis (2004), Mannor et al. (2003), Steinwart (2002, 2004), Zhang